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Abstract 



Although the Federal No Child Left Behind program judges the 
effectiveness of schools based on their students’ achievement status, many policy 
analysts argue that schools should be measured, instead, by their students’ 
achievement growth. Using a ten-year student-level panel data set from North 
Carolina, the authors examine how school-specific pressure associated with the 
two approaches to school accountability affects student achievement at different 
points in the prior-year achievement distribution. Achievement gains for 
students below the proficiency cut point emerge in response to both types of 
accountability systems, but more clearly in math than in reading. In contrast to 
prior research highlighting the possibility of educational triage, the authors find 
little or no evidence that schools in North Carolina ignore the students far below 
proficiency under either approach. Importantly, they find that the status, but 
not the growth, approach reduces the reading achievement of higher performing 
students. The analysis suggests that the distributional effects of accountability 
pressure depend not only on the type of pressure for which schools are held 
accountable (status or growth), but also the tested subject. 




High on the U.S. educational policy agenda is how best to hold schools 
accountable for the performance of their students. One of the goals of any accountability 
policy is to shorten the feedback loops between policymakers, principals and teachers. 
With standards based accountability programs, policy makers set clear standards, 
measure student performance, and use those measures to evaluate the effectiveness of 
schools (Cohen 1996; O’Day and Smith 1993). Successful schools are then typically 
provided rewards in the form of public recognition, financial bonuses for teachers or 
some combination of both. Unsuccessful schools may be sanctioned or provided 
additional support, depending on whether the system is designed to be punitive or 
constructive. The ultimate goal of a standards based accountability system is to generate 
greater student achievement consistent with the standards (Figlio and Ladd 2008; Ladd 
1996). 

With the passage of the federal No Child Left Behind Act (NCLB) in early 2002, 
for better or for worse, student test scores in math and reading have come to represent the 
outputs of interest, regardless of their relationship to any specific curriculum standard, 
and schools are judged primarily on the academic status of their students. In particular, 
NCLB requires every U.S. public school to test all students annually in reading and math 
in grades 3-8 and once in high school, requires each state to set annual targets for the 
percentages of students meeting a state-determined proficiency standard in order to reach 
the goal of 100 percent proficiency by 2013/14, and includes sanctions for schools that 
fail to make the required adequate yearly progress (AYP) toward that goal. In addition, it 
holds schools accountable not only for the overall performance of their students but also 
for that of racial and economic subgroups. Among the many criticisms of NLCB are that 
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the proficiency standards differ across states, the focus on math and reading narrows the 
curriculum, holding schools accountable for annual progress increases the instability of 
school performance measures, more diverse schools are more likely to be penalized, and 
the goal of 100 percent proficiency is unrealistic (Amrien and Berliner 2002; Balfanz et 
al. 2007; Figlio 2005; Figlio 2006; Hamilton, Berends and Stecher 2005; Kane and 
Staiger 2002; Linn 2000; Peterson and Hess 2006). 

Despite these criticisms, many people believe that test-based accountability can 
be a useful strategy for raising student achievement, especially for low-perfonning 
students. The theory of action behind educational accountability is that by setting 
standards and measuring perfonnance relative to standards, teachers will work harder and 
students will leam more. Increasingly, however, observers have argued for shifting the 
metric for school accountability away from the achievement status of a school’s students, 
as is the case under NCLB, in favor of a metric based on students’ growth in achievement 
during the year (Hanushek and Raymond 2005; Ladd and Walsh 2002; Toch and Harris 
2008). 

The argument for using achievement growth rather than achievement status as the 
basis of school accountability is two-fold. First, because children come to school with 
different degrees of readiness to leam and prior achievement levels, many people believe 
it is unfair, and potentially counterproductive, to expect schools alone to offset the effects 
of the background characteristics of their students. Instead, the argument goes, schools 
should be held accountable for outcomes over which they have more control, such as how 
much the children leam during the year, typically measured by their gains in test scores. 
Second, the focus on achievement status, as defined by a proficiency threshold, provides 
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a strong incentive for schools to focus attention on students near the threshold to the 
potential disadvantage of students far below the threshold and of those above the 
threshold. This distributional aspect of status-based accountability programs has received 
significant attention in the recent literature (Ballou and Springer 2008; Booher-Jennings 
2005; Krieg 2008; Neal and Schanzenbach Forthcoming; Reback 2008). At the same 
time, some growth models have been criticized for lack of transparency and their failure 
to require students to meet specific standards. 

The distributional effects of these two types of accountability systems are the 
focus of this paper. In contrast to recent research, which has focused almost exclusively 
on the distributional effects of status programs such as NCLB, we compare the 
distributional effects of a system based on achievement status to one based on 
achievement growth. We are able to compare the two approaches because our empirical 
work is based on longitudinal data from North Carolina where schools have been subject 
to a growth-based accountability system since 1996/97 and since 2002/03 have also been 
subject to the status requirements of NCLB. Given the increasing national policy interest 
in moving to growth models of accountability our comparison of the distributional effects 
of the two approaches is timely. 

Specifically, in this empirical study we use student-level data over time to 
compare and contrast how school-specific pressure associated with the two approaches to 
school accountability affects student achievement at different points in the prior-year 
achievement distribution. The availability of consistent reading and math test score data 
over time allows for careful modeling of student achievement gains, including the use of 
student and school fixed effects to account for time-invariant unobservable characteristics 
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of students such as their ability and schools such as teacher quality. Consistent with our 
theoretical predictions for the status program, we find evidence of positive distributional 
effects for students below the proficiency cut point for both subjects. The response to the 
growth-based program for these low achieving students is more mixed; clear positive 
effects emerge in math but not in reading. The approaches differ further with respect to 
their effects on student performance at the top of the prior-year achievement distribution, 
with the growth approach generating positive distributional effects for these students and 
the status approach either zero or negative distributional effects, with larger negative 
effects in reading than in math. In contrast to many other studies we find no evidence of 
educational “triage” for either program. 

Achievement effects of school accountability programs 

The most convincing analysis of how accountability affects overall achievement 
emerges from cross-state studies such as Carnoy and Loeb (2002), Hanushek and 
Raymond (2005) or from careful district-specific studies that permit comparisons to other 
districts such as Jacob (2005) or Ladd (1999). These and other studies are reviewed in 
Figlio and Ladd (2008). Emerging from research of this type is that the introduction of a 
school-based accountability program generally raises achievement when achievement is 
measured by the high-stakes test used in the accountability system. Some studies also 
report positive achievement effects when achievement is measured by a low-stakes test, 
such as the National Assessment of Education Progress (NAEP) as in Carnoy and Loeb 
(2002) and Hanushek and Raymond (2005), or by a low-stakes state test as in Jacob 
(2005) but in this latter case only in the higher grades. In general, when achievement 
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gains do emerge, they tend to be larger for math than for reading. 



Research has also found, however, that high stakes testing can narrow and 
fragment the curriculum, promote rote, teacher-directed instruction, and encourage 
schools to teach test-preparation skills rather than academic content, tendencies that may 
be stronger in schools with high minority and low income populations (Amrien and 
Berliner 2002; Darling-Hammond 2004; Linn 2000; Nichols and Berliner 2007; Orfield 
and Kornhaher 2001; Valenzuela 2005). Moreover, in schools facing accountability 
pressure, teachers and principals may manipulate the test-taking pool through selective 
disciplinary practices and reclassifying students as requiring special educational services, 
thereby making them ineligible for tests. In addition, and of particular relevance for the 
present study, they may focus instruction and extra resources on those students most 
likely to improve a school’s external standing (Booher-Jennings 2005; Figlio 2006; Weitz 
and Rosenbaum 2007). 

Our main question is how school accountability affects student achievement at 
different points in the achievement distribution in the schools under the most pressure to 
raise achievement. Four distributional questions are of particular interest. The first, and 
most basic, is whether there are any within-school distributional effects, that is, whether 
accountability pressure is associated with greater gains in achievement for students at 
some points in the prior-year achievement distribution than at others. Unless an 
accountability system is specifically intended to change the distribution of student 
outcomes within schools, such distributional effects may well not be desirable. Second, to 
the extent that there are distributional effects, do the largest benefits accrue to students at 
the low end of the distribution? Such an outcome would be deemed desirable provided 
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the goal of the accountability system were to raise the achievement of low-achieving 
students in low-perfonning schools, but less so if the goal were to raise achievement 
across the board in such schools. Third, to what extent do any gains to students at the 
bottom of the distribution come at the expense of those at the top in schools under 
accountability pressure? Fourth, is there evidence of educational triage, in the sense that 
additional resources are focused on students around a designated threshold to the 
detriment of those far from the threshold? Of particular concern is that students at the 
very bottom of the achievement distribution may be so far below the threshold that they 
are worse off under the accountability system than they otherwise would be. 

Several recent studies use different methodologies and data sets to address one or 
more of these questions in the context of status based accountability systems that measure 
school success by student passing rates. Receiving most attention in the literature is the 
issue of “educational triage.” 1 Booher- Jennings (2005) provides qualitative evidence 
from a single school and its associated urban school district in Texas that teachers do 
indeed respond to incentives to increase pass rates as one would expect, namely by 
focusing additional attention on students near the passing rate. Based on careful 
quantitative analysis, Neal and Schanzenbach (forthcoming) document that the 
introduction of two separate accountability systems in Chicago induced schools to focus 
on students near the middle of the achievement distribution to the disadvantage of the 
students at the two tails of the distribution, while Krieg (2007) reports similar findings for 
Washington State. In contrast, in their quantitative study of NCLB in 7 states based on 

1 Although many sorts of triage processes are theoretically possible, we will adopt this term to mean 
“triaging out' ’ students well below and well above grade level and “triaging in” students close to grade 
level. 
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test data from the Northwest Evaluation Association, Ballou and Springer (2008) find 
little or no evidence of adverse effects for the lowest performers. “ 

More generally, Ballou and Springer (2008) find evidence of gains to students at 
the bottom of the distribution, but find no consistent evidence that schools facing 
accountability pressure neglect their high achieving students to focus on low achievers. A 
study by Reback (2008) based on Texas data during the 1990s also generally finds 
positive effects on the very low achievers. Contrary to the triage hypothesis, Reback finds 
that when a school has a realistic chance of improving its accountability rating, the lowest 
performing students make greater than expected gains, even if they have no chance of 
passing the exam in that subject. In addition, Reback uncovers some intriguing 
distributional differences by subject. His evidence suggests that schools respond to 
incentives related to math in ways that increase the perfonnance of low perfonning 
students with at most small adverse effects on higher achieving students. In reading, by 
contrast, except in certain cases, school-wide incentives to raise student performance on 
the reading exam appear to hann students who have a moderate to strong probability of 
passing the exam. These patterns, Reback suggests, may reflect differences in the subject- 
specific strategies schools used to improve perfonnance. When they are under pressure to 
raise math scores, they may well improve basic math instruction for all students, but 
when they are under pressure to improve reading scores, schools may tend to pull 
students out for individualized or small group instruction. 



2 Burgess et al (2005), in a study of accountability in England also finds adverse effects of accountability on 
the lowest perfonning students. In contrast to the studies mentioned in the text, Burgess et al focus on 
perfonnance at the high school level and introduce the element of accountability through school 
competition. 
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Our research makes a three-fold contribution to this literature. First, in addition to 
testing the triage hypothesis suggested by some of the existing studies in the context of 
the status approach to accountability, we compare and contrast the distributional effects 
of the status and growth approaches to accountability. Second, following Reback, we 
compare distributional effects in both math and reading. Third, the fact that we are able to 
match the test scores of individual students as they progress through school means that 
we can use student fixed effects to control for the unmeasurable time-invariant 
characteristics of students, such as their ability or motivation, that might otherwise 
confound the analysis. 

In the following two sections, we first use a simplified model to predict the 
distributional effects of stylized versions of the two approaches to accountability and then 
describe the two programs that form the basis of our empirical work. In the following 
sections, we describe our data and results and end with a concluding discussion. 

Predicted Distributional Impacts of the Two Approaches 

We examine here the incentives faced by teachers (or other school personnel) in 

schools subject to each of the two fonns of accountability. The status approach, as 
epitomized by NCLB, sets a target rate of proficiency, where the target is defined as the 
percentage of students in a particular school and grade level who are deemed proficient. 
The growth, or value-added approach, sets a target for the average rate growth of student 
achievement during the year. Under either system, school personnel in schools that reach 
the specified target in the particular system may receive rewards — financial or 
reputational or both — and those in schools that fail to reach the target are subject to 
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some form of penalty, whether in the form of naming and shaming, external intervention 
and loss of autonomy, or potential job loss. 

In the absence of either type of accountability pressure, we start with the 

3 

following very simple achievement model: 

(1) A lt = A, t _i + u lt 

where A; t is the student’s achievement in year t normalized by the mean and standard 
deviation for all students in that grade in the state. Similarly A u .| is the student’s 
achievement in the prior year, also expressed as a normalized variable for that year and U; t 
is a random error. Thus in the absence of an accountability system the student in this 
simple model is assumed to remain at the same point in the performance distribution as 
she was in the previous year, plus or minus a random error. 

The Status Approach 

With the introduction of a status-based accountability system in which students 
are expected to reach a specified proficiency standard, say P H in year t, students fall into 
two main categories — those with expected achievement in year t above the standard and 
those below (see figure 1). A school that does not expect to meet its overall school target 
rate without additional effort must decide how much additional effort to exert and on 
behalf of which students. 

Provided the school has a relatively large number of students — large enough so 
that the expected value of the random components of the performance of individual 
students is close to zero — the school has little or no incentive to invest additional effort 

3 This model is in the spirit of that presented in Neal and Schanzenbach (forthcoming) but differs by its 
explicit reference to prior year achievement rather than student ability. The use of prior year achievement 
makes the conceptual model consistent with our empirical specification discussed below. 
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in students for whom the expected level of A; t exceeds Ph- For individual students for 
whom the expected level of A; t falls short of Ph — that is, those students with a prior year 
test score below Ph — the school has an incentive to invest up to the point at which the 
extra cost of the additional effort is just equal to the expected benefit to the school in 
tenns of a reduced penalty. As emphasized by Neal and Schanzenbach (forthcoming), 
there could well be some students at the bottom of the expected performance distribution 
for whom the additional effort on the part of the school would simply be too costly 
relative to the benefits for the school to make the additional effort worthwhile. In that 
case, the school would focus its additional attention on the students expected to be below 
the proficiency level, but not so far below to make the standard out of reach. 

Two factors are particularly relevant for determining which students receive 
additional attention — and hence are likely to exhibit achievement gains — in the context 
of this accountability regime. The first is the level of the proficiency standard. The higher 
is the standard, the more likely it is that students at the bottom of the distribution will be 
too far below the standard to make it worthwhile for the school to exert greater effort on 
their behalf. Analogously, a lower proficiency standard, such as Pun figure 1, provides 
incentives for the school to focus attention on students in the lower part of the 
distribution, and the less likely it is that students at the bottom will be “left behind.” The 
second factor is the nature of the educational production function. The easier it is to raise 
student performance at the bottom of the distribution, perhaps through improved 
teaching, tutoring programs or grouping strategies, the greater is the incentive for the 
school to invest additional effort in students whose expected achievement is low relative 
to the standard. 
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Thus, the status model generates one clear distributional prediction. Students 
whose expected achievement is below the proficiency level will receive more attention — 
and hence should achieve at higher levels than they otherwise would have — than those 
above the proficiency level. Less clear is whether there will be a group of students at the 
very bottom who are left behind because of the high costs of raising them to the standard. 

Also not fully clear is what will happen to the achievement of students whose 
expected achievement slightly exceeds the proficiency standard. The presence of the error 
term in expression (1) means that the school has an incentive to devote some additional 
attention to such students; without additional attention, some of them could well fall 
below the proficiency level. The more difficult it is for a school to predict how well its 
students will do, the more likely it is that the school will devote additional attention to 
students just above as well as to students below the proficiency standard. 

For students whose expected achievement is well above the proficiency standard, 
in contrast, the question becomes whether they will receive less attention — and hence 
will achieve at lower levels than they otherwise would have — in the presence of the 
accountability pressure. If additional effort for the students at the bottom is redistributed 
from students at the top, achievement of the higher performing students would fall. If the 
school is able to garner additional resources or find ways to use existing resources more 
effectively than in the absence of the accountability regime, any achievement gains at the 
bottom of the distribution need not come at the expense of those at the top. Thus, the 
impact of a status based accountability system on the high achieving students is an 
empirical question, which depends on how resources are used within the school. 
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The Growth Approach 

The incentives differ when accountability is based on the school’s average growth 
in student achievement. Once again, a school under pressure to improve has an incentive 
to invest additional effort on behalf of any individual student up to the point that the 
benefits of that investment in the form of penalties avoided are just equal to the costs of 
that investment. In this case, however, it is difficult to predict which students will benefit 
most because differential benefits depend on the relationship between additional effort 
and student achievement at different points of the achievement distribution. 

One possibility is that the additional effort needed to raise student achievement by 
a given amount is uniform across students defined by their prior-year achievement. In 
that case, a school under pressure to raise its average achievement growth has no 
incentive to invest any more in one group of students more than in another. Alternatively 
if additional effort generates greater gains for low-performing students than for high- 
performing students — as might be the case, for example, if achievement is measured by 
a test with ceiling effects (that is, one in which the perfonnance of high achieving 
students cannot be distinguished) — a growth-based accountability system would give 
schools an incentive to invest more in the students at the bottom of the distribution than at 
the top. A third possibility is that, consistent with the observation that students at the high 
end of the achievement distribution have made greater gains in the past than those at the 
bottom end, it may be easier to generate larger additional gains at the top of the 
distribution than at the bottom. In that case, schools under pressure would have an 
incentive to invest in the higher performing students, with concomitantly larger gains for 
that group than for other groups. 
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Thus, how a growth based accountability system is likely to affect the distribution 
of achievement gains across students within schools is an empirical question. In general, 
the a priori prediction for large distributional effects is less compelling for a pure growth 
approach than for a status approach to accountability. 

Background on the two accountability programs in North Carolina 

North Carolina is a good state in which to examine the distributional effects of 
these two types of accountability because its schools have been subject to the state’s 
growth-based accountability system since the academic year 1996/97 and the federal No 
Child Left Behind (NCLB) status-based accountability system since 2002/03. Because 
the two systems use different methods forjudging the effectiveness of schools, some 
schools that appear to be performing well under one system may do poorly under the 
other. In addition, in contrast to most other states, North Carolina has long used tests 
that are aligned with the state’s standard course of study, with test scores reported on a 
developmental scale. As a result, the tests measure what teachers are expected to teach 
and students to learn, and students in any grade are less likely to reach a ceiling test score 
than would be the case with a maximum score in each grade. 

The North Carolina ABCs Program 

The North Carolina accountability program — referred to as the ABCs program 
— was part of a broader state effort to improve the academic performance of the state’s 
children throughout the 1990s. First implemented in 1996-97, the ABCs program was 
intended to hold teachers in individual schools accountable for the overall perfonnance of 
their students. Though the program applies to high schools as well, the present study 



13 




focuses solely on schools serving students in grades three through eight. Of particular 
importance for this study, under the ABCs program schools are judged primarily on the 
annual achievement gains of their students from one year to the next. This growth 
approach to accountability was feasible because the state had been testing all students in 
grades three through eight annually in math and reading since the early 1990s — long 
before it was required to do so under the Federal No Child Left Behind legislation of 
2001 . 

From 1996/97 to 2005, an expected average gain in test scores was predicted for 
each student, and the school was deemed effective or not depending on how the actual 
gains of its students compare to their predicted gains. 4 If a school raised student 
achievement by more than was predicted for that school, all the school’s teachers 
received financial bonuses — $1500 for achieving high growth and $750 for meeting 
expected achievement growth. Schools that did not achieve their expected growth were 
publicly identified and in some cases subject to intervention from the state. The intent of 
the program was to induce each school to provide its students with a year’s worth of 
learning for a year’s worth of education. In 2005, the formula for calculating growth was 



4 The expected average gains were predicted as follows. For each grade and subject (i.e. math and reading), 
a student’s expected score is based on an equation of the form TS t - TS t _i = a + bXl + cX2 where TS t is the 
test score in either math or reading in year t and TS t _i the test score in the same subject in year t-1, XI is a 
proxy for the student’s proficiency and is measured as the sum of the student’s math and reading scores for 
the previous year minus the state average, and X2 is designed to account for regression to the mean and is 
measured as the student’s prior year score in the subject of interest minus the state average in that subject. 
The tests are scored on a developmental scale and the parameter “a” can be interpreted as the statewide 
average gain in score for students in the specified grade and for the specified subject. The parameters a, b, 
and c were estimated using 1994 test scores for each grade. Because the b and c coefficients were quite 
similar across grades for each subject area, the state uses a single pair of b and c coefficients for each 
subject area to determine the expected growth rates. For further discussion see Ladd and Walsh (2002). 
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changed, but the focus on holding schools accountable for achievement growth, rather 
than levels, remained. 5 

In addition to their growth rankings, schools also receive various designations, 
such as schools of excellence, schools of distinction, and priority schools, based on the 
percentages of students meeting grade level standards, which carry with them no 
financial bonuses. In addition, some schools are labeled “low performing” based on their 
high failure rates as well as their poor growth performance. Thus the ABCs program does 
not completely ignore achievement status. At the same time, the teachers’ bonuses are 
based solely on the growth in student achievement. The existence of positive incentives 
does not alter the predictions of the simple model presented above. A school’s failure to 
meet its growth standard still imposes costs on its teachers; the cost is simply the bonuses 
foregone. 

No Child Left Behind (NCLB) 

The federal government started holding schools accountable for student 
achievement with the 2001 reauthorization of the federal Elementary and Secondary 
Education Act, called No Child Left Behind. This law applied to schools in North 
Carolina and elsewhere starting in the 2002/03 academic year. NCLB requires states to 
test students annually in reading and mathematics in grades 3-8, and assesses schools on 
the basis of whether their students are making adequately yearly progress (AYP) toward 
the ultimate goal of 100 percent proficiency by 2014. Moreover, each school must meet 

5 The new formula no longer is based on changes in students’ developmental scale scores from one year to 
the next. Instead, it is based on changes in test scores normalized based on the mean and standard deviation 
from the first year a particular test was used in the state. The academic change for an individual student is 
now calculated as the student’s actual normalized score minus the average of two prior year academic 
scores, with the average discounted to account for reversion to the mean. 
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annual proficiency targets not only for the student body as a whole, but also for various 
subgroups defined by race, socio-economic status, and disability within the school. 
Failure to meet AYP brings with it consequences, such as the right of children to move to 
another school and the requirement that districts use their federal Title 1 grants to pay for 
supplemental services, including those from private providers. After five years of failure, 
the school is subject to state takeover by the state, an outcome that, to date, has been rare 
across the country, and is not directly relevant for this study which ends in 2007. 

Under NCLB, North Carolina policy makers must set annual proficiency targets 
— defined in terms of the percentages of students who are at grade level — that will 
assure that each school is on target toward the 2013/14 goal of 100 percent proficiency. 
The result is that under the federal law each school faces an annual target defined in 
terms of achievement status rather than in terms of achievement growth as under the state 
accountability system. 6 Not surprisingly, a school that perfonns well under the state’s 
accountability system may do poorly under the federal system, and vice versa. 

Figure 2 illustrates the variation over time in the percentages of schools failing to 
meet the two types of accountability standards by year. Included in the percentages are 
schools that failed both standards, a percentage that ranged from four percent in 2003 to 
3 1 percent in 2006. The bottom line is that for the past 1 1 years, many schools in North 
Carolina have not met one or both of the standards for student achievement. How a 



6 More recently. North Carolina and several other states have been provided a waiver under NCLB to 
incorporate some elements of the growth model into the federal accountability standards. Under that 
provision, some students who are on track to meet the proficiency standard within three years now 
contribute to a school’s progress toward the goal. Because the growth is still evaluated in terms of progress 
toward the absolute standard rather than in relation to a predicted growth standard, however, the system 
remains essentially a status model, rather than a growth model. 
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